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ABSTRACT 
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Board (HFQMB) was established in 1996 to help Naval Aviation achieve its mishap 
reduction goals. Previous HFQMB mishap reduction initiatives focused on flight 
mishaps and aircrew related issues, however a recent shift to broaden the scope now puts 
attention on aircraft maintenance and consequently maintainers. Baker (1998) developed, 
from a Maintainer’s perspective a Climate Assessment Survey (MCAS). The purpose of 
the MCAS was to provide a diagnostic tool to identify potential intervention areas from 
the perspective of maintenance-related mishaps. This thesis seeks to provide a vehicle 
fostering the proliferation of the MCAS throughout the fleet by adapting it for the World 
Wide Web (WWW), and developing a prototype Web site. A usability analysis was 
conducted to form a basis for future modification to the online version of the MCAS, 
with the ultimate goal being an effective tool to promote the reduction of human factors 
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I. INTRODUCTION 


A. HUMAN FACTORS QUALITY MANAGEMENT BOARD 


1. Overview 


Following the tragic January 1996 Nashville, Tennessee Navy F-14 mishap that 
killed the aircrew and several local residents, Vice Admiral Bennitt, the then Commander 
of the Naval Air Force, U. S. Pacific Fleet, created a Human Factors Quality Management 
Board (HFQMB) to improve aviation safety through identifying strategies to prevent 


human error. The purpose of the HFQMB simply stated in its charter (1996) is: 


To significantly reduce the rate of occurrence of naval aviation mishaps 
caused by aircrew/operator error by identifying systemic improvements in 
the processes and systems that affect human performance and guard 
against human error. Further, to institutionalize continual improvement in 
these areas. 


Senior Naval Aviation leaders recognized a need to form a coalition between them 
as policy makers and individuals who work directly in the Naval Aviation community. 
Consequently, the HFQMB is composed of a Flag Officer and senior operational Officers 
from both the Navy and Marine Corps including a cross section of all communities and 
ranks from Squadron Commanding Officers to Fleet Lieutenants. The implementation of 
the HFQMB findings was intended to foster a reduction in the 1996 Naval Aviation class 
‘‘A” flight mishap rate by 50 percent within three years and by 75 percent within 10 years 


(HFQMB Charter, 1996). 


2. Human Factors Quality Management Board Efforts 


The HFQMB’s strategy of taking an across the board approach to improve its 
processes and systems encompassed a three-prong approach, including: (1) Mishap Data 
analysis, (2) Benchmarking Best Procedures, (3) Climate Safety Assessment (See Figure 
1). Mishap Data Analysis included the practice of observing patterns that may have 
formed in past mishaps, and trends that may been have established in relation to human 
factors. Bench Marking involved observing the practices used in other organizations to 
combat human factor related error. The Climate Safety Assessment was designed to 


qualify issues that might impact these human factor errors. 


Mishap Data 
Analysis 
Recommendations 
for Intervention 
Benchmarking > 3 
Brainstorming 
Climate oe 
Assessment 


Figure 1. HFQMB Methodology 
From “AIRPAC Brief,” by CDR J. Schmidt, March 1998. 


The first area of interest to the HFQMB, analysis of mishap data, represents 


research conducted without the need for external data collection. The Naval Safety | 


Center (NSC) maintains a database of mishaps dating back 20 years; this historical data is 
categorized by 1050 characteristics (NSC, 1999). Taking the NSC data and classifying 
the cause and type of mishaps enabled aggregate analysis by the HFQMB. The NSC’s 
mishap database also allowed the HFQMB to analyze the mishaps with enough specificity 
that when coupled with the appropnate modeling techniques, enabled meaningful action 
development. 

Benchmarking processes/practices was the second area of endeavor by the 
HFQMB for mishap reduction. Benchmarking is the process through which internal or 
external organizations’ performance levels are compared, and its processes/practices are 
observed. The organizational benchmarks can be utilized to establish guidelines and 
ultimately lead to "Best Practice" decisions (Chapman, 1997). In this instance, the 
HFQMB observed processes from organizations that seek the same safety goals as Naval 
- Aviation such as, the U.S. Air Force, and the commercial airlines, etc. (HFQMB, 1996). 

The third area of focus by the HFQMB is addressed by the Command Stele 
Assessment (CSA), a tool designed by Ciavarelli and Figlock (1997) at the Naval 
Postgraduate School to measure safety climate in a squadron from the aircrew 
perspective. The CSA (Discussed in subsequent chapters) enabled the HFQMB to 
measure the climate and process effectiveness within an organizations’ flight operations. 
Recommendations were deemed a critical part of the overall success of the HFQMB’s 
actions. Using the information acquired through the means of observation and collection, 
the HFQMB priontized the recommendations in terms of their potential to achieve the 
mishap reduction goals and forwarded them to the appropriate authority for coordination 


and implementation at the organizational level of Naval Aviation (HFQMB, 1996). 


3. Human Factors Quality Management Board Results 


Historically, the mishap rates of the last two decades have been low by past 
standards (See Figure 2). However, despite a low mishap rate, it is important to realize 
that Naval Aviation strikes several aircraft per year. An example of the effect lost aircraft 
have on the fleet is illustrated by the data taken from fiscal year 1997 through 1998. In 
FY °97, a total of 65 aircraft were accepted into the Naval Aviation inventory (Naval 
Aviation News, 1998). During FY ‘98, 36 aircraft were removed from the inventory as a 
result of mishaps, (NSC, 1999). The aircraft lost in 1998 represent 55 percent of those 
purchased in the previous year. The class “A” mishap rate has been essentially on a 
plateau since 1988. Without a change in leadership action or the conduct of Naval 
Aviation, the mishap rate would not be expected to deviate from its current level. 

Contributing to mishap data analysis, Shappell and Weigmann (1995) adapted 
Reason’s (1990) Human Error Model to re-classify the divisions of Naval Aircrew Flight 
mishaps. This classification model took the larger category of human error and expanded 
it to three sub-areas of responsibility (Nutwell & Sherman, 1997). The new divisions of 
human error consisted of: unsafe aircrew conditions, unsafe supervisory conditions, and 
unsafe aircrew acts. The redesigned model gave a better understanding of the human 
cause factors than previous classifications in the traditional model; more accurate 
reporting resulted in improved understanding, and clearly indicated the appropriate 
intervention areas to achieve the goals sought by the HFQMB. Notably, in a report issued 
in 1997 by the Department of the Navy, 40 percent of the procedure violations committed 
by aircrew were conscious breeches, and 50 percent of the supervisory failures were 


labeled unsafe supervisory conditions. Both aircrew and supervisory conditions are an 


indication of climate within the organization. Relatively little work has been completed 
on organizational factors when considering the effectiveness of a squadrons performance 
and safety (Ciavarelli & Figlock, 1997). Program and policy of the squadron are a part 


the safety climate and culture maintained within the organization. 


Figure 2. FY 50-99 Flight Mishap Rates. 
NAVAL AVIATION MISHAP RATE 
FY 50-99 
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From: Naval Safety Center 1999. (Through 30 JUN 99) 


The efforts of the HFQMB were expected to bring about fundamental changes in 
the way business was conducted in Naval Aviation, with these changes, the established 
mindset regarding attainable mishap reduction would have to change. (HFQMB Charter, 
1996). The recommendations of the HFQMB resulted in a record mishap rate for the 
U.S. Navy in 1997, with a mishap rate of 1.29 %, the lowest level of class A mishaps ever 
(NSC, 1999). The U. S. Marine Corps reached a comparable milestone in 1998 with the 
organizations lowest mishap rate of 2.52 % (NSC, 1999). The past actions taken by the 
HFQMB have had the desired effect of reducing the human factor related mishap rate. To 


continue to reduce the human factors related mishaps, refinements in the process 


evaluation would need to take place. Nutwell and Sherman (1997) state human error 
represents a large portion of the causal factors that contribute to Class “A” mishaps and 
propose additional reductions in the mishap rate could be attained with refinements in this 
area. Traditional approaches targeting aircrew human factors should be expanded to 
include the maintainer with potential gains in mishap reduction. If further improvements 
in mishap reduction were to occur continued exploration of all facets of prevention 
needed to be studied and understood. Efforts in mishap reduction were concentrated in 
the aircrew side of human factors caused mishaps, expanding the focus to include the 
maintenance caused human factors mishaps could enhance the overall reduction of the 
mishap rate. The desire to measure the climate of safety from a maintainer’s viewpoint 


was a topic that necessitated the development of a tool for this purpose. 


B. AVIATION SAFETY CLIMATE ASSESSMENT 


1. Command Safety Assessment 


The Command Safety Assessment (CSA) was developed by Cuiavarelli and 
Figlock (1997) of the Naval Postgraduate School; its goal was to estimate the 
effectiveness of organizational safety programs. To assess the effectiveness of safety 
programs and processes, it is necessary to understand the climate that is maintained 
within the organization. Organizational culture is defined by Merntt and Helmreich 


(1996) in the following manner. 


Culture can be defined as the values, beliefs, rituals, symbols and 
behaviors that we share with others that help define us as a group, 
especially in relation to other groups. Culture gives us cues and clues on 
how to behave in normal and novel situations, thereby making the world 
less uncertain and more predictable for us. There are two important and 
distinct components of culture. The surface structure, or outer layer of 
culture consists of observable behaviors and recognizable physical 
manifestations such as member’s uniforms, signs and logos, and 
documents. The deep structure, or inner layer of culture, consists of the 
values, beliefs and assumptions which underlie the surface structure and 
provide the logic which guides the members’ behaviors. 

Measuring culture and looking at the safety climate an organization fosters is part 
of understanding the organization and the people that make up the group. Each person 
brings their history and multi-cultural memberships with them to an organization (Merritt 
& Helmriech, 1996). If organizational culture has an impact on the safety of the group, it 
would be desirable for the group to have a unified, integrated organizational culture, 
rather than one made up of diverse values. Much attention has been focused on human 
error in complex systems, but relatively little has been accomplished on the potential 
contnbution that organizational factors have on crew performance and safety (Ciavarelli 
& Figlock, 1997). 

Work in the area of climate measurement within Naval Aviation squadrons is 
built on research conducted by Roberts (1990). Roberts discussed the attributes in 
organizations that were successful in reducing nsks associated with hazardous operations. 
Her label for this type of organization was “high-reliability’ and described an 
organization that worked in a hazardous environment, yet produced very low rates of 
accidents. The CSA study was based on a model of high reliability organizations, and 


produced a questionnaire that would provide feedback to the HFQMB conceming the 


issues of command climate, morale of Naval personnel, workload, resource availability, 


and estimated success in safety intervention programs. This newly developed tool, the 
CSA, could be used by the Aviation Safety Officer (ASO) to assess the organizations’ 


effectiveness regarding their commands safe operation (Nutwell & Sherman, 1997). 


3. Maintenance Climate Assessment Survey 


Safety culture within an organization effects the entire output of the group. Naval 
Aviation is recognized as a high-risk organization, devastating consequences occur if 
something goes wrong within any aspect of the organizations operation. Just as the 
concept of the CSA was created as an extension of the principles Paine the study of 
high reliability organizations (Ciavarelli & Figlock, 1997), the Maintenance Climate 
Assessment Survey (MCAS) became the maintenance focused extension of the CSA 
(Baker, 1998). The MCAS was developed with the notion that operational climate and 
maintenance climate hold the same importance within the unit, therefore, both contribute 
to safe completion of the organizations mission. With maintenance actions contributing to 
the mishap rate, the need to modify a survey tool to address the issues pertaining to 
maintenance became relevant (Baker, 1998). 

The MCAS has a focus on safety climate from the aviation maintainer's 
viewpoint. Baker's (1998) goal was to develop, administer, and validate a questionnaire 
to assess the effectiveness of Naval Aviation maintenance operations in the management 
of msk. Baker concludes that the developed prototype MCAS with a few noted 
exceptions conformed to the existing Model of Organizational Safety Effectiveness 
(MOSE) quite readily, he successfully adapted the model to make it maintenance specific. 
Statistical analysis allowed the onginal 67 question MCAS to be reduced to a more 


practical length of 35 questions. Refinement and consideration for distribution of the 


MCAS offer areas of further development. 


C. RESEARCH OBJECTIVE 


The objective of this study is to further enhance the effectiveness of the MCAS 
developed by Baker at the Naval Postgraduate School (Baker, 1998). This study creates 
an online version of the MCAS, and incorporates this version of the MCAS within an 
electronic distribution system and tests the usability of the online system. The intent is to 
increase the availability of the MCAS, enhancing present efforts to reduce maintenance 


related mishaps and the overall mishap rate. 


D. STATEMENT OF PROBLEM 


Utilizing the distnbution capability that is created by the World Wide Web, an 
online version of the MCAS can be distnbuted to an ever-increasing audience with 
virtually no increase in distribution expense (Lewis, 1998). To further reduce the 
maintenance attnbuted aviation mishaps, safety process and programs that contnbute to 
safe operation will need to improve. Not simply seeking improvements in process and 
programs, the aviation community sought the invention of new processes to enable the 
desired improvements in mishap reduction (Nutwell & Sherman, 1997). 

Distribution of information on the World Wide Web offers the opportunity for the 
Naval Aviation community to “invent” the new process that it desires, bringing about 
changes in safety and ultimately reducing the mishap rate. As Lewis (1997) states, the 
World Wide Web offers increased exposure to an audience without increases in 
distribution costs, and as a distribution medium, the World Wide Web can disseminate 


information that was originally designed for distribution in other forms. Specifically, the 


MCAS was created to be distributed in paper form, with modification to format, the 
MCAS can be reconfigured to World Wide Web specifications. Distribution of the 
developed MCAS by an effective Web based method will enable more organizations to 
use this tool for self-evaluation. 

The main thrust of this study focuses on the effective reconfiguration and 
distribution of the existing MCAS through the World Wide Web. Usability of the online 
version of the MCAS was the measurement of effectiveness. Understanding and 
measuring the effectiveness of the safety program is vital to the reduction of human factor 
caused mishaps. It is essential to provide tools to the aviation organization for self- 
assessment of the effectiveness of their processes and programs. Four issues are relevant 
in the discussion of an electronic distribution of the MCAS, specifically, this thesis 
explores the following questions: 

1. Is there a need to disseminate and administer Web based materials to aviation 

units? 

2. Do U.S. Navy and Marine Corps squadrons have the necessary 

software/hardware capabilities to utilize the World Wide Web? 

3. Can the MCAS be effectively distributed and administered using the World 

Wide Web? 

4. Once distributed, do personnel have the necessary ability to utilize the 


electronic version of the MCAS? 


E. SCOPE AND LIMITATIONS 


The intent of this study is to gain a better understanding of the distribution 


capability the World Wide Web offers organizations with the proper hardware and 
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software in place. It is essential to determine the usability of an online version of the 
MCAS for use by Naval Aviation squadrons. Subjects from the Naval Aviation School 
of Safety will be used due to their close proximity, this effort represents a preliminary 
effort to measure the usability of the online version of the MCAS. An exit survey will 


allow the users to rate their impression of this method of MCAS distnbution. 
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Il. LITERATURE REVIEW 


A. OVERVIEW 


The review of the literature considers three areas pertinent to the effective online 
delivery of an organizational climate assessment tool. Areas covered include: 
1) Organizational climate and its assessment to provide background on the traditional 
process; 2) High Reliability Organizations (HRO) as a model for effective organizational 
climate 1s considered and its application to the military; 3) Development of the Command 
Safety Assessment (CSA) and the Maintenance Climate Assessment Survey (MCAS) for 
Naval Aviation based on the HRO model; 4) The software development process entailed 
for organizational assessment tool delivery; and 5) The Human computer interface 
concept, functionality and usability. The information is summarized and considered in 


context of the research questions. 


B. ORGANIZATIONAL CLIMATE AND ASSESSMENT 


Daft (1989) defines organizational climate as: 


...the set of key values, guiding beliefs, and understandings that are shared 
by members of an organization. It defines basic organizational values, and 
communicates to new members the correct way to think and act, and how 
things ought to be done. It represents the unwritten, feeling part of the 
organization. (p. 503) 
Climate is the unseen part of an organization that helps members make sense of 
organizational events and activities (Daft, 1989). When an organization enjoys 


widespread agreement, climate within the organization is strong, conversely, if little 


agreement exists within the organization, climate is weak. Daft (1989) also notes that 


le 


leadership plays a large role in the organizational climate. He states that simple mandates 
of policy do not form an organizations cultural climate, values and beliefs come from 
observing leaders within the organization. Climate is an important feature of the 
organization, and if a change in organizational climate is desired, leadership must 
implement more than simple policy to effect the needed changes (Schein, 1992). 

To understand more fully the process of change, it is helpful to study the needed 
elements that will successfully facilitate it. Burke (1994) looks at organizational 
development as it relates to change. Much work in this area is built on a model 
developed by Lewin (1958) called the three-step procedure of change. The first step of 
this model is unfreezing the present level of behavior, which 1s no more than reducing 
prejudice and clarifying the area that requires change. The second step is movement, take 
action that will change the social system from its original state of behavior. The final and 
third step is refreeze, or establish processes that will secure the new behavior from 
change. 

Schein (1992) elaborates on Lewin’s model, enlarging and defining the scope of 
each step. He further sets forth useful assumptions when studying an organization and 
attempting to measure culture. Schein notes, it 1s important to keep in mind that the 
contextual meaning of cultural assumptions can only be fully understood by the members 
of that culture, so creating a vehicle that helps the members of the group understand their 
own culture is actually more important than the researcher gaining this perspective. Since 
culture is a shared set of assumptions, obtaining initial data in a group setting 1s 


appropriate. 
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C. HIGH RELIABILITY ORGANIZATION (HRO) 


In research conducted by Roberts (1990), and extended by Libuser (1994), an 
HRO is defined as an organization that operates in a hazardous environment, yet produces 
very low rates of accidents and incidents. HROs seem to have less than their “fair share”’ 
of failures despite operating in a complex, demanding environment involving significant 
risks and hazards. Roberts and Libuser note that these HROs had certain characteristics 
that made them organizationally effective managers of nsk. Libuser (1994) went on to 
develop a Model of Organizational Safety Effectiveness (MOSE) that categorized five 
traits that were common among this type of organization, these traits include: process 
auditing, reward systems, quality of operations, nsk perception and command control. 

HROs are found in many areas of manufacturing and everyday life (Perrow, 
1984). The operation of power plants, oil refineries and transportation represent a few 
industries where accidents are rare but if they occur have catastrophic and far reaching 
effects (Reason, 1998). The operating concepts of HROs can apply to many other 
industries where accidents can not be tolerated, the food industry, drug manufacturers, 
financial institutions are just a few of the organizations that characterize groups that 
depend on high reliability. Finally, the HRO concept can also be extended to military 
applications (Roberts, 1988). 

Reason (1998) descnbes a theory of active and latent causes of accidents and 
applies these causal factors to HROs. An active failure is one that is directly attributable 
to an action or violation by an individual directly influencing a system. Latent conditions 
may exist for years within a system and not cause an accident until the proper sequence of 


events come together simultaneously. He contends the perception of individuals within 
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an HRO 1s that iron clad defenses are in place preventing accidents, however in practice, 
organizational defenses are not iron clad, and may have certain weaknesses. When the 
weaknesses in separate defenses occur together in the proper sequence, failure may result 
(See Figure 3). Meeting peak requirements or time pressures may cause individuals 
within the HRO to neglect the ongoing process of finding holes in the accident prevention 


Strategy; infrequent mishap occurrence contributes to this mindset (Reason, 1998). 





Asset Loss 


Figure 3. Perceived and Actual Defenses from Risk 


Research conducted by Roberts (1990) and Libuser (1994) indicate that the 
complexities of systems today necessitate new strategies for prevention, the simple direct 
causation of an accident by one individual does not fit a HRO. HROs must continually 
seek new methods of process improvement and not become complacent despite the fact 
they suffer from few accidents (Perrow, 1984). The perception that these organizations 
maintain “iron clad” defenses against hazards it can not hamper continual process 
improvement (Reason, 1998). Strategies to achieve a HRO would include a strong 
hazard reporting system with no fear of retribution by the members, as well as 


maintaining a flexible, learning culture within the organization. 
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Bierly and Spender (1995) adapt the HRO model concept for military application 
and claim that a nuclear powered submarine fits many of its assumptions. They observe 
culture interacts with the formal organization, and that the strong organizational culture 
within the nuclear submarine supports the formal structure. Roberts (1988) finds an 
increasing number of organizations in modem society fit the HRO model, observing the 
size and complexity of the aircraft carmer organization coupled with a seemingly low 
incidence history in dealing with high risk evolutions qualified it as a HRO. Baker (1998) 
states, “In high-risk endeavors such as aircraft operations, it is essential that such support 
be flawless, as the price of deviations is high.” Naval Aviation as a whole with its 
complexity, high risk, and low tolerance for error fits the criteria to be a HRO. 

Building on the work of Roberts and Libuser, Ciavarelli and Figlock (1997) 
tailored the MOSE to fit the conditions found in Naval Aviation. They created the CSA, 
a tool designed to assess the safety climate within a Naval Aviation squadron from an 
aircrew perspective. The goal of this research was to develop and apply a methodology 
that enabled a command to assess it performance on key safety issues, providing a 
feedback tool as part of the mishap prevention process. Planned follow on efforts include 
cross-comparing squadrons, aircraft type and demographic variables in an attempt to 
construct an empirical foundation for assessing hazards and nisk. 

Baker (1998) furthered the efforts of the CSA by designing and validating the 
MCAS that would accomplish the same objectives as the CSA, but from the perspective 
of the aviation maintainer. With the help of maintenance experts, he took a list of 
maintenance topics and created 67 questions specifically tailored to aviation maintenance. 


Baker’s initial 67 question survey through factor analysis was further refined and reduced 
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to a 35 question instrument. The MCAS was successfully proven to support each of the 
MOSE components. 

Currently, Naval Postgraduate School Operations Research Department students 
are engaged in taking the MCAS and validating it by analyzing the results within an 
aircraft community, and across aircraft communities (B. Goodrum, Personal 
Communication, 1999; T. Oneto, Personal Communication, 1999). To further enhance 
the effectiveness of the MCAS, this study seeks to provide a wide reaching delivery 
system on the World Wide Web. Effective distribution will involve the creation of an 


electronic delivery system, and the usability testing of that system. 


D. SOFTWARE TOOLS 


The software necessary to carry out the successful development and 
administration of the online MCAS includes two categories: developmental software, and 
user software (Alexander & Tate, 1999). The developmental software in this research- 


created both the Web site, and also the online version of the MCAS. The user software is 


the tool each user must employ to utilize the online MCAS once developed; this software 
is determined by the end user and is a World Wide Web browser. Both categories of 


software and its employment are discussed below. 


1. Developmental Software 


Perseus Survey Solutions for the Web is the software that was used for the 
transformation of the MCAS to an online version of the same. The Perseus Corporation 
makes this software product Microsoft Office compatible (Strebe & Perkins, 1998). 


Compatibility with Microsoft products allows for future expansion of the online MCAS 


to include functions contained in Microsoft Access and Microsoft Excel. 

The most common application of this development software is construction and 
distribution of surveys, but its other functionality includes the development of inquiry 
forms, feedback questionnaires and order forms (Perseus Survey Solutions User Guide, 
1998). Functioning as a word processor, Survey Solutions for the Web acts as an 
interface between the plain language text formulation and translation to Hypertext 
Markup Language (HTML). Once textual data is converted to HTML, it may be posted 
on a server and users may access the stored data through their computer and an Intermet 
browser. The storage and availability of these files is further described in the discussion 
of the World Wide Web. 

Data collection is the final function that the Perseus software package carmes out 
as part of the design of the online MCAS. Once collected through the user response to 
questions within the online MCAS, data is packaged in a standard tab separated form 
(ASCII TSV) which may be further stored, or interpreted using Microsoft Access or any 
program that allows the ASCII TSV data entry (Perseus Survey Solutions User Guide, 
1998). 

The second software development tool used was Netscape Navigator Gold 
Version 4.5. This software package was used in the development of the Web site that 
presented the online MCAS to the user. Netscape Navigator Gold utilizes a powerful 
What- You-See-Is- What-You-Get (WYSIWYG) HTML generator that uses many of the 
same functions found in word processors (Authoring Guide, Netscape Navigator Gold, 
1995). Maintenance Aviation Safety material was gathered from existing sources and 


combined with original material to form a distribution vehicle for the MCAS. 


ee 


2. User Software 


The final piece of software used in this project is the Internet Browser (browser). 
A browser is defined as “a computer program that enables the user to rad HYPER-TEXT 
from disk files or on the World Wide Web” (Downing, Covington & Covington, 1996). 
A browser is the software each end user must utilize to access the online MCAS, and 
becomes the most dynamic portion of the all the software packages used. The dynamic 
nature of Internet Browsers 1s discussed more fully in subsequent sections and will be an 


ongoing concer in a Web based project. 


E. WORLD WIDE WEB 


1. Overview of the World Wide Web 


The World Wide Web is a unified, interconnected interface to the vast amount of 
information stored on the computers around the world (Sobell, 1995). Information can be 
displayed on the World Wide Web using a combination of text, graphics, animations and 
audio to form dynamic documents. Once created, these documents are displayed on a 
user computer by an Internet browser. A browser provides easy access to Web sites that 
are posted on many servers connected to the World Wide Web. A series of Clients, 
computers that receive information from other computers, and Servers, a computer that 
provides services to another computer, make up the Internet. 

Composed of HTML code, a Web site is made up of one or more pages to create 
an organized presentation of material through the use of Hyperlinks. The Authoring 
Guide, Netscape Navigator Gold (1995) defines the Web site as a package that contains; 


topics that can be linked to from other Web sites, may require the use to scroll the browse 
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window to see the entire page, can contain links to graphic files and other pages, and 
finally can be located on a server so that other users may access the page. Below, Figure 2 
illustrates the composition of a Web site that is made up of several Web pages, connected 
by use of Hyperlinks. 

Hyperlinks are the mechanism that allow an entire Web site composed of many 
Web pages to be connected in a logical sequence. The use of Hyperlinks to tie together 
multiple Web pages of the same theme allow a user to navigate a rather complex Web site 
with ease and little confusion. The World Wide Web offers and opportunity to distnbute 


information to a vast audience at very little cost (Lewis, 1998). 
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Figure 4. Overview ofa Web site 


2. The Internet Browser 


The browser is an integral part of using any online software applications, 


including the online MCAS. Each browser interprets HTML code by applying a set of 


IL) 


WebSite 


defaults established by the individual user. A default is a set of assumptions that a 
browser makes unless it Is given instructions to the contrary (Downing, Covington & 
Covington, 1996). These defaults within the browser determine how the browser 
interprets the HTML code and displays the information on the users monitor. Different 
interpretations of HTML code result in different views of the same information. For this 
reason, two browsers may interpret the same HTML code differently. In the design of 
Web sites, it becomes necessary to consider the different default settings users will utilize 
and anticipate the effects this may have on data within a Web site application (Randall, 


1999). 


3. The Client/Server Model 


The Client/Server model is the basic model that enables individual users to access 
a Web site at any given time (Sobell, 1995). The Server is a program that runs all the 
time waiting to receive a request for the information it contains in its data stores or 
memory. World Wide Web clients are the computer users who access a server through 
the use of a browser. The Client/Server interaction occurs using a computer language 
known as Hypertext Transfer Protocol (HTTP). HTTP is a computer language that works 
across all platforms; consequently, whether the Client and Server are the same type of 
computer does not inhibit the ability for the two computers to communicate (Downing, 
Covington & Covington, 1996). When the Client requests information from the Server 
via HTTP, the request is acknowledged and data is sent back to the Client. (See Figure 5) 

The Client/Server Model is responsible for enabling the efficiencies in 
distribution of online material throughout the World Wide Web (Tanenbaum, 1992). 


Once posted on a Server, the Online MCAS will incur the same distribution cost as it 
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reaches an ever-increasing audience. The efficiency in online distribution is analogous to 
the infinite shelf space retailer of the frictionless economy (Lewis, 1997). The ability to 
distribute training matenals at a small, fixed cost, to an audience that is potentially 
infinite represents a method of improvement for training that may facilitate reaching 


mishap reduction goals set within the organization of Naval Aviation. 
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Figure 5. Illustration of the Client/Server Model 


F. HUMAN USABILITY ISSUES 


Hawkins (1998) observes that “Human factors is about people...It is about their 
relationship with machines and equipment, with procedures and with the environment 
about them.” He illustrates this relationship through the SHEL model, where the elements 
of Software (e.g., training, procedures, etc.), Hardware (e.g., controls, displays, etc.), 
Environment (e.g., temperature, lighting, etc.), and Liveware (e.g., operator, systems, 
etc.), pair to form a human system interface. Liveware 1s considered the hub of the model 
and all remaining components must be effectively matched to this central component. 
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Human factors principles have traditionally been applied to hardware devices such as 
monitors, keyboards, etc. as well as workstation design to optimize their fit with human 
needs and capabilities (Grandjean, 1987). The proliferation of hardware and software 
technology necessitated the same intensive application of human factors technology to 
enhance software Human Computer Interface (HCI) design (Brown, 1989). Today, most 
software and design applications integrate HCI design concepts including: fonts, color, 
etc., as well as screen layout and functionality. 

The “browsers” that the World Wide Web is accessed through have gone passed 
many HCI design iterations to maximize their efficiency and ease of use (Frederickson- 
Mele, Levi, & Conrad, 1999). Web software and that used in Web site development have 
also been crafted using HCI concepts with similar objectives as in software development. 
Using the Internet as a distnbution method raises unique design issues that are dependent 
on user terminals and pre-selected defaults, consequently, World Wide Web applications 
benefit from the HCI research in the browser they are viewed through. This unique set of 
user circumstances requires that Web sites be tested for usability by their intended 


audience. 


1. Usability Testing 


Software and its design presents many challenges to the user in trying to 
accomplish a given task. A reasonable goal of any software project should be the ease of 
use of the final product (Jordan, 1998). Usability issues are not only important within the 
realm of consumer products, but also those products used within a commercial or 
professional context (Jordan, 1998). Past studies indicate, with the exception of safety 


testing, usability testing takes place in the worst possible arena, the marketplace (Randall, 
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1999). In other words, Randall (1998), states whether a product was usable or not was 
left up to the end user, after the purchase or attempted use of the final product. Formal 
usability testing became recognized about a dozen years ago in the computer software 
industry. Usability testing does not speed the design process up, but if user satisfaction is 
the goal, the increased development time is well spent. Constructing a site to meet the 
needs of your intended audience increases the impact the overall presentation has upon 
the user (Frederickson-Mele, Levi, & Conrad, 1999). 

Delivering a prototype of the product to a test group followed by a questionnaire 
is a method Jordan (1998) found beneficial in determining the attitudes and expectations 
of potential product users. Jordan specifies that data obtained from usability studies 
should answer the following questions: 

e Are there usability faults in the product? If so, what are these faults? 

e How can these usability faults be solved? 

e What are user attitudes towards the use of this product? 

e What design changes are required to make the users’ attitudes more positive? 
Open ended questions, allow the respondent to write their own answers to questions and 
prove particularly useful in determining what the important issues were to the users. A 
usability study enhances the effectiveness of modification to a prototype, ultimately; the 


final product will have more utility to the end user (Jordan, 1998). 


2. Writing For the Web 


When writing for Web sites, studies have shown that users do not actually read: 
instead, they scan the text (Nielsen, 1994). This finding suggests a style of wnting that 
can be absorbed through scanning rather than through in-depth study. During Nielsen’s 
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four years of usability testing, he considered site architecture, navigation, search, page 
design, and style yielding surprising results. When asked to comment on the Web site, 
users critiqued content quality and relevance to a much greater extent than navigational 
issues or page elements (Nielsen & Sano 1994; Nielsen, 1997). This suggested content 
would play a very large part in the users perception of the Web site. Morkes and Nielsen 
(1997) suggest three main content oriented conclusions be drawn from their study. 
1. Users do not read on the Web; instead they scan the pages, trying to pick out a few 
sentences or even parts of sentences to get the information they want. 
2. Users do not like long, scrolling pages: they prefer the text to be short and to the 
point. 
3. Users detest anything that seems like marketing fluff or overly hyped language 
and prefer factual information. 
It is clear that users require concise presentations of content when on the Web, and have 
very low tolerance for superfluous words or ideas in the material. 

Morkes and Nielsen (1997) state that wnting effectively for the Web is a function 
of sensitivity to the users needs balanced with the message you wish to covey to the user. 
Writing in a concise, scannable style would seem to fit the users need for quick fact 
finding, while ensuring content was available to the user. It was noted that when a page 
comes up, the readers attention shifts to center screen and the body of the text was read 
before they looked over headers or other navigational elements that made up the page 
(Morkes & Nielsen, 1997). Similar findings on the importance of content resulted from 
the work of other similar studies of online materials (Spool, Scanlon, Schroeder, Snyder, 


& DeAngelo, 1997). This indicates the need to maintain a concise writing style on the 
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Web page, paying particular attention to the relevance of the material presented to the 


intended user. 


G. SUMMARY 


An understanding of organizational climate leads research in the direction of 
categorizing Naval Aviation as a HRO. As a HRO, Naval Aviation safety programs and 
processes can be analyzed and improved using the CSA and MCAS. Working within the 
confines of current distribution methods, the WWW offers unique opportunities for 
widespread, inexpensive distribution of safety material. Special consideration must be 
given to design issues regarding usability of software applications. A vehicle to distribute 
the online version of the MCAS will further the need to assess safety program 


effectiveness in Naval Aviation. 
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It. METHODS 


A. RESEARCH APPROACH 


Naval Aviation seeks methods to improve its process and programs regarding 
safety in an effort to reduce mishap rates. The MCAS developed by Baker (1998) was 
designed to be utilized by squadrons to assess the effectiveness of their safety programs 
from a maintainers perspective, making it possible to measure the safety climate within 
an aviation command. Making the MCAS available to more aviation units through 
effective distribution on the World Wide Web is the goal of this study. 

To measure the effectiveness of the online MCAS, a prototype Web site was 
developed and distributed to a sample user group. Each subject tested the Naval Aviation 
Maintenance Safety Web site with a prepared task list that required them to navigate 
through the pages of the Web site. At the completion of the task list, the subject would 
have viewed all the pages of the Web site, and presumably formed an opinion on its 
effectiveness. Subjects upon completing the task list filled out an exit survey composed 
of demographic questions of their computer background and perusal of the prototype Web 


Site. 


B. DATA COLLECTION 


1. Subjects 


Students attending the Aviation Safety Officer Course (ASO) at the School of 


Aviation Safety served as subjects in the study. Selection to the ASO course 1s made by 


individual Aviation Commands and yields a cross section of the Naval Aviators and 
Naval Flight Officers from all aircraft communities within the U.S. Navy and Marine 
Corps. Squadron ASOs are responsible for the safety programs in all squadrons and will 
likely be one of the primary users. Each subject from this test group will have a direct 


influence on the safety climate within their squadrons. 


2. Apparatus 


The Naval Postgraduate School’s School of Aviation Safety provides a computer 
lab for the ASO students during their training. Each Officer is issued a password that 
allows computer usage through a group account. The computers in the lab each had a full 
functioning prototype of the Aviation Maintenance Safety Web Site. After a subject 
gained access to the computer through use of the group account, the icon “Aviation 
Maintenance Safety Web site” resided on the computer desktop. Once the Web site was 
opened, it appeared as any World Wide Web site, although this prototype was installed on 
the hard drive of the computer rather than on a server on the Internet. The prototype was 
kept locally on the lab computers to facilitate changes that would occur in its 
development. Microsoft Internet Explorer was the user software provided to the subjects. 
All of the functions within Internet Explorer operated in the same manner as if the 


application was taken from the World Wide Web. 


3. Instrument 


The exit survey consisted of three parts: 1) Demographic items or computer 
experience, 2) Likert type questions assessing feeling towards the Web site, and 3) Open- 


ended items to elicit subjective opinions. The first part, collecting demographic 


30 


information, allowed the subject to choose from a list of descnptors (e.g. Squadron level, 
Intermediate level, etc.). The second and third part of the survey was a combination of 
Likert style questions and open-ended questions that allowed the subjects to interject 
comments they felt were relevant. The Likert questions were constructed using a five 
point rating scale with verbal anchors, as follows: Strongly Agree, Agree, Neutral, 
Disagree, and Strongly Disagree. The open-ended questions were included in the 
questionnaire to elicit respondent viewpoints regarding such factors as the overall 
impression of the Web based distribution, and comments regarding improving the 
usability of the Web site. The open-ended questions provided subjects an opportunity to 
comment on aspects of the prototype Web site they felt were not adequately covered in 


the previous portions of the survey (See Appendix D for survey). 


4. Procedure 


The subjects were divided into two groups of 26 and presented with an overview 
presentation of the study. The overview consisted of a projected computer demonstration 
of the Aviation Maintenance Safety Web site, and distribution of the materials necessary 
to carry out the user test. The materials used to direct the subjects exposure to the Web 
site included: Instructions for viewing the Aviation Maintenance Web site (Appendix B); 
Aviation Maintenance Safety Web Site Task List (Appendix C): and User’s Impression of 
the Maintenance Safety Web Site and MCAS (Appendix D). The Instructions for 
viewing the Aviation Maintenance Web Site was a guide for the subject to follow 
enabling him/her to log on to the group account provided in the computer lab and access 
the Web site installed on the computer. As a portion of the group overview, specific 


questions subjects expressed on using the computers in the lab were addressed. The Task 
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list was designed to take the subject through a series of planned navigation routes within 
the Web site. By completing the task list, the subject would have navigated through the 


entire prototype of the Aviation Maintenance Safety Web site. 


C. DATA TABULATION 


The data was transcribed from the exit survey into the software program SPSS 
(SPSS v. 9.0). Multiple choice responses were each assigned a value from one through 
four. The Likert style questions were based on a five point scale, these questions were 
coded into the SPSS software using five (5) through one (1) corresponding to the 
respective anchors (Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree). 
Once the data was transcribed, the analysis tools in the software SPSS were used to 
generate descriptive statistics including the mean, standard deviation, range, and 
frequency distnbution. Content analysis was conducted on the responses provided from 


the open-ended survey questions. 


D. DATA ANALYSIS = 


1. Analysis Strategy 


The data from the surveys was transferred to the statistical analysis program SPSS 
(SPSS v.9.0) for analysis and interpretation. One-way Analysis of Vanance (ANOVA) 
testing matched the level of user experience, Independent Vanable (IV), with selected 
Dependent Variables (DV) gathered in the survey. The one-way ANOVA was used to 


assist in answering the difference (group comparison) issues raised from the study. 


By 


IV. RESULTS 


A. FREQUENCY AND CONTENT ANALYSIS 


The 21 item exit survey was administered to 52 subjects with a response rate of 
88% (n = 46). The test users represented a cross section of the aviation commands that 
make up the squadrons of the Fleet. Each aviation command is required to have an 
officer trained by the School of Aviation Safety; as a result, the entire fleet is represented 
in this school at any given time. The sample included flight qualified Officers from Naval 
and Marine Corps Aviation and did not include Non-Commissioned Officers or Enlisted 


personnel. 


1. Demographic Information 


The material collected in Part I, the demographic portion of the exit survey 
determined the experience level each subject had with computers, as well as, the Internet 
and Internet browsers. This information is later used to determine if experience level 
with Internet browsers affected a subject’s level of satisfaction, and impacted the usability 
of the Web site. Question one on the survey revealed that all participants were members 
of aviation units that performed maintenance at the squadron level (n = 46, 100%). 
Question two asked, “How long have you been using an Internet browser?” Table | 
shows the breakdown regarding experience each subject had with Internet browsers. A 
total of 82 percent (n = 38) of the subjects had at least one year of experience using the 


Internet. 
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Table 1. User Experience With a Computer Browser 







2. User Satisfaction With the Web Site 


Part II of the exit survey examined the subject’s impressions of the layout and 
content of the Web site. Questions 6 through 9 specifically targeted the subjects’ overall 
impression of the Web site. Question 6 asked, “My tour of the Web site was very 
interesting.” and the mean was 3.87 (SD=.686). The range for question 6 was 4, and the 
histogram of the frequency distnbution is presented in Figure 6. Clearly many 
respondents (74%) agreed that the tour of the Web site was interesting, with only one 


person disagreeing with this statement. 
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Figure 6. “Interesting” 
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Question 7 stated, “I found the Web site easy to navigate” and the mean was 4.35, 
(SD = .822). The range for question 7 was 4, and the histogram of the frequency 
distribution is presented in Figure 7. The vast majority (91%) agreed with this statement, 


while three persons disagreed. 


Number of Responses 





Strongly Agree Neutral Disagree Strongly 
Agree Disagree 


Figure 7. “Easy to Navigate” 


Question 8 asked, “I feel the information on this Web site was in a logical form.” 
and the mean was 4.16, (SD = .556). The range for question 8 was 3, and the histogram 
of the frequency distribution 1s presented in Figure 8. Most respondents (91%) agreed 


that information was logically represented, with only four persons neutral on this topic. 
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Figure 8. “Web Site in Logical Form” 
Question 9 asked, “The information presented in the Web site is relevant to 
maintenance safety” and the mean was 4.37 (SD = .488). The range for question 9 was 2, 
and the histogram of the frequency distribution is presented in Figure 9. All subjects 


(100%) either strongly agreed or agreed with this statement. 
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Figure 9. “Information Relevant” 


After completing the Likert style questions in the first section of Part II, the 


respondent was asked to comment on his/her impression of the site through the open 
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ended question, “My overall impression of the Web site is: Below are listed some of 
comments that typified the response to this question. 

“Looks like it'll be useful. Especially the MCAS and MCAS analysis tools.” 

“Simple, straight forward and should have useful information for safety issues.” 

“Very user friendly Web site which contained both interesting and useful 

information.” 

“Should be a good tool for promoting safety and existing tools.” 

Subjects expressed general enthusiasm for the Web site and thought it would be useful in 
the distnbution of safety related material. 

The next four questions (10-13) of the exit survey looked at consistency issues. 
Consistency 1s important in both term usage and implicit instructions, this makes 
navigation through the Web site possible with minimum confusion. Whether consistency 
is achieved was determined by the impression of the respondent. These four Likert style 
questions were posed to allow each user of the prototype to comment on their perceived 
consistency throughout the exercise. 

Question 10 asked, “During my tour of the Web site, | found the use of terms 
consistent” and the mean was 4.05 (SD = .630). The range to question 10 was 4, and the 
histogram of the frequency distribution is presented in Figure 10. The majonty of users 


(74%) agreed with this statement, with only two users neutral and two disagreeing. 
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Figure 10. “Use of Terms Consistent” 


Question 11 continued with consistency by asking, “During my tour of the Web 
site, | found The position of screen instructions were consistent” and the mean of this 
question was 4.31 (SD = .510). The range to question 11 was 3, and the histogram of the 
- frequency distribution is presented in Figure 11. The majonty (98%) agreed with this 


statement, and only one user was neutral. 
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Figure 11. “Screen Instructions Consistent 
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Question 12 asked, “I found the results obtained from the Web site were 
predictable” and the mean was 3.94 (SD = .611). The range of question 12 was 3, and the 
histogram of the frequency distribution is presented in Figure 12. A majority of the users 
(78%) found that they received consistent results from the Web site, although, nearly on 


quarter of the respondents (22%) were neutral on this aspect. 
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Figure 12. “Results Predictable” 


Question 13 asked, “During my tour of the Web site, I found adequate instructions 
were provided to navigate” and the mean was 4.22 (SD = .757). The range for question 13 
was 4, and the histogram of the frequency distribution 1s presented in Figure 13. Most 
users (89%) aggreed adequate instructions were provided, while three persons were 


neutral and two persons disagreed. 
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Figure 13. “Adequate Navigation Instructions” 


The next five questions on the survey (14-18) addressed the organization and 
clanty of the information presented in the prototype Web site. These questions 
concentrate on the experience of the user while learning the methods employed to 
navigate the site. An open-ended question at the end of this section provided the subject 
an opportunity to voice opinions not covered in the Likert scale questions. 

Question 14 asked, “I feel that the information on the Web site is easy to 
understand” and the mean was 4.16 (SD = .868). The range for question 14 was 4, and the 
histogram of the frequency distribution is presented in Figure 14. While a majority 
agreed (87%) the information was easy to understand, two subjects were neutral, with two 


disagreeing. 
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Figure 14. “Information Easy to Understand” 


Question 15 stated, “I feel that it was easy to learn to use this Web site” and the 
mean was 4.42 (SD = .652). The range for question 15 was 3, and the histogram of the 
frequency distribution is presented in Figure 15. A large majority (91%) agreed with this 


statement, while half (50 %) strongly agreed with the ease of learning the Web site. 
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Figure 15. “Ease of Learning the Web Site” 
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Question 16 stated, “J feel that the organization of material on this Web site is 
clear” and the mean was 4.05 (SD = .815). The range for question 16 was 4, and the 
histogram of the frequency distribution is presented in Figure 16. Most users (82%) the 


organization was clear, while five users were neutral and three users disagreed. 
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Figure 16. “Organization of Material” 


Question 17 stated, “I feel that I could effectively use the information on this Web 
site” and the mean was 4.35 (SD = .566). The range for question 17 was 3, and the 
histogram of the frequency distribution is presented in Figure 17. The majority of 


respondents (96%) agreed, with two persons being neutral. 
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Figure 17. “Effectively Use the Information” 


Question 18 stated, “J feel that when fully implemented, this Web site will have 
all the functions and capabilities I would expect it to have” and the mean was 4 (SD = 
.760). The range for question 18 was 4, and the histogram of the frequency distnbution is 


presented in Figure 19. Most users (81%) agreed, whereas nine users disagreed. 
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Figure 18. “Web Site has all Functions I Expect” 


The open ended question in the final section of Part II of the Exit Survey asked, 
“To improve the usability of this Web site, | would.” The purpose of this question was 


twofold; first, allow the respondent to voice any usability concerns that were not covered 
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in the Likert style questions, and second, highlight any issues the study may have omitted. 


The responses to this question are summarized in Table 2. 











‘To improve the usability of this Web site, I would” 


Type of Response Number of Responses Percent of Total 


Table 2. Comments on Usability Improvements 


Of the 30 responses, most commented that the Web site was user fnendly and did 
not offer suggestions for improvement to the prototype. The responses that suggested 
changes to the prototype were typified by the comments below. 

“Provide more links on the first page to make it more efficient and easier 

to the final page I am after. Index or reference at the start or bottom of the 

page would work.” 

“Perhaps on the opening slide there could be something like a table of 

contents. It would be useful to know everything that is contained on the 


Web site.” 


‘“... Also, home page is confusing, I would title the button START HERE 
instead of ENTER.” 


“Add a link to the NASA safety site.” 
“Use Frames.” 
A common theme of these answers was one of mild navigational preferences, some 


relating to verbage and others related to the physical layout of the Web site. 


3. User Satisfaction With the Online MCAS 


Part III of the Exit Survey contained questions pertaining to the online version of 


the MCAS. To effectively use the online MCAS, the user needed to be capable of finding 
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the MCAS in the Web site, and he/she also needed to comprehend the method of 

completing the questionnaire and submitting it. Questions 19 through 21 deal with the 

feelings about the clarity of instructions and ease of completing the online MCAS. 
Question 19 asks, “Finding the online MCAS in the Aviation Maintenance Safety 

Web site was straight forward” and the mean was 4.44 (SD = .655). The range for 

question 19 was 4, and the histogram of the frequency distribution is presented in 

Figure 19. The vast majority of users (96%) either strongly agreed or agreed with this 


statement, and two were neutral or disagreed. 
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Figure 19. “Finding the MCAS” 
Question 20 asked, “The instructions for completing the MCAS are easily 
understood” and the mean was 4.64 (SD = .488). The range for question 20 was 2, and 
the histogram of the frequency distribution is presented in Figure 20. All respondents 


(100%) either strongly agreed or agreed to this question. 
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Figure 20 “Instructions for MCAS Understood” 


Question 21 asked, “The task of putting answers in the MCAS was easily 
completed” and the mean was 4.55 (SD = .585). The range for question 21 was 3, and the 
histogram of the frequency distnbution is presented in Figure 21. Mainly users (96%) 
strongly agreed or agreed with this question, with the exception of two persons who were 


neutral. 
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Figure 21. “Submitting Answers to the MCAS was Easy” 
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The final question in Part IIT of the Exit Survey was the open ended question “I 
would make these changes (if any) to the administration of the online MCAS.” The 


response rate to this question 1s summarized in Table 3 below. 









“T would make these changes (if any) to the administration of the online 
MCAS.” 


Type of Response Number of Responses Percent of Total 
| NoResponse | 12 | 26% 


Table 3. Comments on MCAS 


Content analysis of the open-ended responses yielded a grouping of similar 
themes. Users commented on the content of the survey as well as the administration, 


below are typical responses to the online MCAS. 


“The survey 1s too long... Your average trooper will not willingly (hence 
honestly) fill out such an in-depth survey.” 


‘Tama Marine 53-E Pilot and many of the questions did not pertain to me. 
Some of those did not have N/A as a selection.” 


‘“E-6 Community- Where would it fall?” 
Administrative Comments: 


“Have an option of no graphics/cookies. Units at sea have minimal 
bandwidth and download time is terrible.” 


‘T tried to tab thru [sic] the questions (instead of mouse). | thought hitting 


ENTER would record an answer, but instead... it submitted the survey 
before I was complete.” 
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. having to manually scroll down continually to answer more questions 
was annoying.” 


“Have more questions on one page” 


Comments in this area ranged from a concern that the length of the survey might 
inhibit the validity of some respondent’s answers (attention span) to physical 
limitations of the movement through the Web site (utilizing back button of 
browser, etc.). 

The final area of the Exit Survey was labeled “Comments” and posed two the 
open ended questions “The most negative aspects of this Web site were” and “The most 
positive aspects of this Web site were”. The response rate to each of these questions is 


summarized in Table 4 and Table 5. 


“The most negative aspects of this Web site were.” 


Type of Response Number of Responses Percent of Total 


eee [Te 


Written Response 





Table 4. Comments on Negative and Positive Aspects of online prototype 


“The most positive aspects of this Web site were.” 


Type of Response Number of Responses Percent of Total 





Table 5. Comments on Negative and Positive Aspects of online prototype 
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The Comments on both negative and positive aspects of the prototype Web site 


were analyzed and the most representative comments are listed below. 


Negative Aspects: 


‘Having to back out after completing the survey” 


‘Marine billets not included” 


“Lack of visual stimulation for the Gen X/ Mountain Dew crowd.” 


‘Too much wasted space on the main pages” 


“Survey might be too long, you’ll surpass the attention span of your target 


audience.” 


‘Survey on the long side” 
Positive Aspects: 
‘Ease of use, simplicity” 


‘Power point Lectures” 


‘A great information hwy for maintenance. Nothing but positive results 


can come from this site.” 


‘Generally easy to use” 


‘Easy access, easy to navigate, accessible to everybody” 
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‘Should provide good background data” 


The negative comments centered on a concern that young sailors would not take 
the time to complete the survey because of it’s length. The positive comments 
characterized a Web site that was easy to use and contained useful information for 


safety programs. 


B. ANOVA TESTING 


1. One Way ANOVA 


SPSS software provides a metric called the significance level, also known as p 
value, which allows the researcher to interpret a calculated statistic in the generated 
output without using a critical value table. The symbol p equates 2 the probability of a 
Type I error. A Type — is the likelihood of rejecting the null hypothesis when it is 
actually true. If the reported p is less than .05, the finding ts statistically significant and 
the null Hypothesis (no difference) ts reyected (Morgan & Gnego, 1998). 

A one-way ANOVA was used to compare the means of different respondent 
experience levels to test for significant differences in levels of usability. ANOVA 
assumes the Dependant Variable (DV) 1s interval scale, normally distributed, and the 
variance of the groups are equal. If the assumptions of ANOVA are violated, an 
alternative nonparametric test called the Kruskal-Wallis (K-W) test should be used to 
compare the differences. It was not necessary to use the Kruskal-Wallis test with the data 
from this study, since it ~ all the parameters necessary to for ANOVA analysis. The 
Independent Variable (IV) in this study was represented as ordinal data while the DVs 


were measured on an interval scale. The distributions of each of the DVs were compared 
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to anormal distribution curve to ensure they approximated the normal distribution. 

The equality of vanances was tested using the Levene Test (Kotz, Johnson, & 
Read, 1988). Since the Levene statistic was not significant (p >.05), the assumption of 
equal variances was not violated and the K-W test was not needed. (Morgan & Griego, 
199s). The ANOVA Statistic compares means between groups and 1s used to identify a 
Statistically significant difference between means. If the p value calculated by an 
ANOVA test 1s significant (p < .05) then there is a significant difference between groups 
with respect to the DV. In order to answer the difference between the IV and the 


individual DVs, hypotheses were developed and tested. 


2. Formulation of The Null Hypothesis 


The null hypothesis is used for testing; 1t 1s a statement that no difference exists 
between the parameter and the statistic being compared to it (Cooper & Emory, 1995). 
The Alternative hypothesis is the logical opposite of the null hypothesis. If statistically 
significant, the null hypothesis can be rejected, and the alternative hypothesis 1s accepted. 
The following hypotheses formed and tested to compare how the IV (User experience 
level) impacted the DV (User impressions). 

1) IV: How long have you been using an Intemet Browser? 

DV: I found the Web site easy to Navigate. 

NULL HYPOTHESIS: there is no difference in how expenence level impacted 

the ease of navigation of the Web site. 

The F statistic with 3, and 42 degrees of freedom was 1.494. This resulted in a p value of 


.234, which did not allow rejection of the null hypothesis. 
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2) IV: How long have you been using an Internet Browser? 
DV: I feel the information on the Web site was in a logical form. 
NULL HYPOTHESIS: There is no difference in how experience level impacted 
perception of whether the information was iiceal in a logical form. 
The F statistic with 3, and 42 degrees of freedom was .799. This resulted in a p value of 
.503, which did not allow rejection of the null hypothesis. 
3) IV: How long have you been using an Internet Browser? 
DV: J found that adequate instructions were provided to navigate the Web site. 
NULL HYPOTHESIS: There was no difference in how experience level impacted 
whether respondent felt enough instructions were provided. 
The F statistic with 3, and 42 degrees of freedom was 1.494. This resulted in a p value of 
.234, which did not allow rejection of the null hypothesis. 
4) IV: How long have you been using an Internet Browser? 
DV: It was easy to learn to use this Web site. 
NULL HYPOTHESIS: There is no difference in how experience level impacted 
the ease of learning to use the Web site. 
The F statistic with 3, and 42 degrees of freedom was 2.287. This resulted in a p value of 
.097, which did not allow rejection of the null hypothesis. 
With each case, the null hypothesis could not be rejected. The compamnson of the 
subject’s experience level with Internet browsers to usability concerns, does not impact 


the impression the subject had on the Web site’s usability. 
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V. CONCLUSION 


A. SUMMARY 


Naval Aviation is posed with the challenge of reducing its mishap rate, making it 
essential to pursue programs and processes to achieve this. Organizational climate has 
been linked to mnsk management and, consequently, experienced mishaps. Understanding 
the climate that pervades an organization is essential if changes are to be made to 
ultimately effect the mishap rate. 

Many theoretical frameworks have been developed to understand organizational 
climate; one that fits the characteristics of Naval Aviation is the High Reliability 
Organization model (HRO). Working within the HRO framework, the Command Safety 
Assessment (CSA) survey evaluated the safety climate within Naval Aviation Squadrons 
from aircrew perspective. The Maintenance Climate Assessment Survey (MCAS) was 
designed to measure HRO characteristics, but from a perspective of the aviation 
maintainer. 

In order to provide for an effective vehicle to distribute the MCAS, the World 
Wide Web (WWW) was explored. The WWW offers an opportunity to obtain a wide 
distribution of material at a very small, fixed cost. Shnnking budgets within the military 
necessitates finding new methods of mission accomplishment, therefore making the 
efficiency of the WWW a method of allowing process improvement within the confines 


of shnnking budgets. 


B. CONCLUSIONS 


This research, based on the initial research questions, has given the following: 

1) Is there a need to disseminate and administer Web based materials to aviation 
units? Naval Aviation is challenged to meet aggressive reductions in its mishap rate by 
the year 2000. Even with the record low mishap rates that have been achieved in recent 
years, continued improvement is necessary. The HFQMB is seeking the “invention” of 
new processes and procedures to help meet the goals set forth in mishap reduction. A 
Web based MCAS will further facilitate climate assessment and evaluation throughout 
the Naval Aviation Fleet, improved methods of assessment will lead to process 
improvements for Naval Aviation. 

2) Do the U.S. Navy and Marine Corps squadrons have the necessary 
software/hardware capabilities to utilize the WWW? As technology permeates the fleet, 
it is clear that the ability to utilize the WWW at the squadron level exists. A total of 97 
percent of the subjects in this study had access to the WWW (n = 46). The remaining 3 
percent involved in this study were not familiar with the computing capability at their 
future squadron assignment. 

3) Can the MCAS be effectively distributed and administered using the WWW? 
Development software available enables the design and implementation of surveys for 
applications on the WWW. The HTML editors that are commonly available facilitate the 
design of a Web site for distnbution of the MCAS. The current level of technology can 


easily support the dissemination and collection of data from the MCAS. 
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4) Once distributed, do personnel have the necessary ability to utilize the 
electronic version of the MCAS? The subjects in this study were representative of ASOs 
throughout the fleet, 82 percent (n = 46) had over one year experience using the Internet. 
Successfully navigating the prototype of the Naval Aviation Maintenance Safety Web 
Site, subjects not only demonstrated the requisite knowledge for using the online tool, 
they demonstrated an acceptance and enthusiasm for the project. 

Adequate knowledge exists within the ASO community to make effective use of 
the WWW as a distnbution method for the MCAS. Using the efficiency obtained 
through the WWW allows increased distribution of the MCAS with minimal cost. 
Taking advantage of the software and hardware that is currently in place within aviation 
squadrons is an additional step to achieving the improvements that Naval Aviation must 


accomplish to meet its goals of mishap reduction. 


C. RECOMMENDATIONS 


Many areas remain open for further development and testing of the Naval 
Aviation Maintenance Safety Web Site. It is suggested that further research efforts 
include the following areas: 1) Inclusion of survey feedback into the layout and design of 


the Web site, 2) Populating the pages that were data space holders, (e.g., summaries of 


mishaps, etc.), and 3) Further development of data capture methods. 
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APPENDIX A. AVIATION MAINTENANCE SAFETY WEB SITE 


This appendix consists of three parts: (1) A verbal descnption of the 
pages that make up the Aviation Maintenance Safety Web site, (2) A symbolic 
depiction of the logical layout of the Web site, (3) Screen captures of the pages 


that make up the complete prototype Aviation Maintenance Web site. 


Part 1. Verbal Description of Web Pages 
Page 1: Title page for the Web site. 
Page 2: Welcome/Purpose page. Branches to page: 2.1 
Page 2.1: Safety Survey Page. Branches to pages: 2.1.1, 2.1.2, and 2.1.3 
Page 2.1.1: MCAS 
Page 2.1.2: Fleet Statistics on Excel 
Page 2.1.3: Interpretation Materials 
Page 2.2: Related Web sites. 
Page 3: Aviation Maintenance Training Page. Branches to pages: 3.1, 3.2 
Page 3.1: Groundcrew Coordination Training. Branches (0 pases: 3p 32 ands 
Page 3.1.1: Workshop Matenials. 
Page 3.1.2: Student Guide. 
Page 3.1.3: Case Studies. 


Page 3.2: FAA Document. 


Page 4: Community Maintenance Human Factors Bnefs. 


Page 4.1: Community Human Factors Bnefs for Jets. 
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Page 4.2: Community Human Factors Briefs for Props. 


Page 4.3: Community Human Factors Briefs for Helos. 
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Part 2. Symbolic Depiction of the Logical Layout of the Web Site 


Page 


Page 


: Page 
22 





Page Page Page 
B.1.] Sale2 33 
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Part 3. Screen Captures of the Aviation Maintenance Safety Web Site 





File Edit View ae Commies Hee: eee 


ee Et bi Ran tend Me ee ee oe er ah ee ee aS 


2 , ; 4 ; 


Back Fouad pes! Home Search poe Print Security oo 


— 2 Bookmarks A Location: [file ///D\/sample functioning website/avsatwebpage/aaa start here. htm -| Whe 
| 7 Instant Message a Internet cs Lookup cs NewéCoo! e RealPlayer 


School of Aviation Maintenance Safety 


oe” a 


Law 








Monterey. C alifornia 





Page 1 
Title page for the Web site 
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<< fleet stats and excel - Netscape 
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Edit View Go Communicator Help =] | | : 


* 





% ers ve —a = ne “o> 4 Aa or A 3 @ Seer —_ . a = — owe” ay) ee %A os . ET a Le ° ae ——x y's 4 ee 
i . = »- aad S.A a ' ; : ss : m ; a 

°3 amin y we < roo. oar eee : fy 4 - ° = 5 

3! Back raseec. Reload. Home. Search Netscape Print. Secumtp-  Ste> 








a ee oe 


aE Bbokmarks - Kh Locatorr | mple functioning website/avsafwebpage/Groundcrew coord pwrpt presentations. htm 7 a Wha''s Related ‘ 


Ee ee So ge le ee 





ae eens = seta 


DA Pe arti paella (ee ae 

| EX Instant Message S Intemet [iy Lookup Esj New’£ool 

Fee Ops ST Ny i Sore eae Sieeer es 

ane: eee geen Se te ak See Spake ass : 
wie : E r 


Rtg ny eae eae a peo SPE RSet ot @ kare? ‘ 
ed ht hoe EWC ita p Werpol 
~ pA! oS 3 be ae 
a ae aan : 
. " id ie 





age TO APO gr ee er 8 Stee = 
*, 
+ 


Fg a eae 
one = ia ~ te a wes cor 
Peigtis nee he 


PY ER RE EN ees 
=e. that canb 
. Ye "hee ae awe n 


FEES 


Ry eo 2 4 f x py; “Py a ee 
7 eA EN BAe BERS. ~ ie eae Ce SORE 

es anes re CME om f 30: ye citi a ke + yn ee Te 8 3 
I EL LAS Dt Dire ae FE8 Ss OE Aes BO COLTER IE AL Poh OLE 





Page 3.1.1 
Workshop Materials 


69 


Fie Gdit View Go Communicator Help © 


TR TN ma ee Tm n 





| 
i 
< 
< 
3 
i 


SOTA 7 DOI ERY er) ye eee F 
oe et ole eat i en 


la” aa ras f 
a Pal (eval 7 > 4 


|Search. Netscape 
Sonn file: ///F l/sample functioning 
o4 Lookup 


Reload. Home. 


- ae F Loc 
. s an Tak 











” 


















re | New?Gool 





2 B 

7 
tA) ae dy re 44 
SEie, BFE Betas = 
rahe? 


a - yay 
ROE. Specs 


hye re Ores 


Purpose cece of tre ane dudes vad 
TO Wore Onis ase Ceeer people, 


Dow ‘clock s0@ bur | duet dar you cas 
read ts.9 suff, 1019 we Mipporsed to be 
eraayles of type 


Rucr ame gomy o sellyof tall. Docu 
Coma at ie bere 9 dae ANEDOS af 


a femme . hat does thabave O40 
about asutneg. 


Tuatss@ do sce ! dow cthak thar thee 
wil be the cad Cafe reaping if I dow't 
lowe aoe aneg Ofer aca. Tesss6 
ceat of he Cametrgeety brad away 
symem. | cow asd oe Oat you Cas 
fag © Ceep you mast ti9 ae tC domsg 
toa. 


Pace: sele2 







S “+ . . 3 2 
pean 4 2g Tae ro mrermy ea % =] me) ; BYE i Tag ¥ gue ee Me ota et OO 00 me a PONG 4 
F : 4 Listing Uy eo 


eae ee an ee 









4 : ee e ad 
CRC BPRS EPO LAE pA ILD 








areas ne 


Student Guide 


70 


BgTy 


— Search Netscape Prt Senn) 
7S Sree 5 Bookmarks & Location: file: ///D\/sample functioning « webste/avsalwebpage/case Studies of mishaps htm 


nie El ae C4 Loc D cS New&Cool SD pear 


Eater Son 


] EI" wets Related 


= == 





{x aor car 


Case studies of Sicendcr related mishaps 





~ ~~: rate 


Mishaps listed by community or date 


FP SHIFTER Re WOH” PH AE Pa OER "DOE 


rt IETS aS > EE 

marie 

AD OOS OY DDB eow im 

L mth te Ren tee Lt. LS eepep  s 
TOME WON BY Ct 

A 6 OU Orbe 141 

hi OE Ls Le ose 

dott Co") oe hee Cae 

rk? or +coore & t~ ar 


Fu... OF 
D> eA Ne et aE I 8S MBVSie8 8B 


cern 
trio Xx wun DT Um T ann 
AP & EFBQ we BOA FT Oe8 (lush Ae Ott Wee's 
8 re 
TA Ee OO EAT } OIF TAD CD 1908 O14 AID HO re 
rr 


PORRF ks ODI SIAL ee Ame WO PUR DWT LOS OR ee A ROS Moe TE 
Rae SO eu cs ube 

secon EA BE or O19 19 GY 1K 

ae BPs Met +) Oo 


on? Cele ses 

3D 68 OO OTe were ota pei BP Ow. Bee HRD 

L4A1a7er: ae? bt de te RD ee ee ee ~ bf) 

eer CUA tk Lee 

Ae ole eS AN ARO Oe! ORY PRE Nhe Pas OR Ae AA URS TOA Oh tet 8 a A, OF 
SaTLECiLy AGA HID 

Z LIN 

OTPERS TESTIS COPEETTTEECETIROTOVECIETER CO ER ITT ECC IES) 

IAA 08 

CAO VE 2.998 EE BOP 2 et) oer 44a 8 Be HOP 8) i 

Mere 

OL SED UTI 38 COMUIC Pr 

VRAD DALI 


TdTE° BI G9GE' DS 
@.2: rae 


oer CPF SHE AE Pe TA" oe 
Fe SOLOS ST (ta The OS. BAO Ew 
rw etre Nernst A, OD 
enn OTe DCU MN 
ee ee KING Sv UR AE. KE 
PwWASE A 
Pooa I. acTGs FL E Bret ewesrres therm 
ee ore 
Ad MOE LEIS tots 62 tA, rs of) Beak PT ew 


ON imdb acid To 


Meret VTS Tht Ua ..., BTM 


FD VERO AE HED ep een Ee BT 

Uawe:.f, G4 240 

FD POPS A. OS 3 LA GOP | SVE S Ott 9 OB 3-15 
Vat inte Lyne 

CS FOUR E wD 

ROT RL 2°) ET BAR! SERS Mere FE IL HS OMe 





Paces: lee 
Case Studies 


7) 


a d 2 
- « . ‘ ‘ 


E . 


= 


er Fi ; > as — _ 
iL . il 2 
, : rs ' 
EAS Pye: Ne eet a oe 3 n . “a: ota 

‘i ; — mc Ac fe . oat : . ole oy il mein 

Pe ‘ ’ - = )-? ‘Ss cme, 

et ~ F i] g = 

a é me. a oe 
in, 


Pett Men NA rea ce | 


Poe 2! RTH MAD 
Pir - i ae 


| ca “Sooke 
Taisen Bl 


een yaa me gree Te Prerrets oa Fe RRR FOU MR HN TNR mE (OO CER A Lae Og Wem at ity my MRR Hp (iets G merry Meant ae ern etm ome pen ty ee mg 
ics ; . -— ; fs cau ; h : 
. o> comp 


- 
Intemet C44 Lookup alee ee a ieee 


a 
Ai BIKE", es : 9 my ae 


Ease 


en hy 
; Fg: Cn es rat 
ca? ahve ‘ = ts, 
BETO. Pea es 
Oo: fy" 3%, ~~~ = bys s . 
< : RN ne ae 


2 


Purpose cace of me ane duces bed 
TO wiv mids ae Cebeer provie, 


Cos ¢tloct 10% buc | Guak Gat you ces 
read Ges auf, 1012 pm Migporerd & be 
emuapies of type 


Eber ss he gomep o gollyof cal. Coc 
Giagt Gare bere Sea anance a& 


WS fisem , Seat does es bave dd 
about anus ag. 


Er ene te ae Ae einem 8 EK I tte wwii a at ee mt are at tat a he nies ty AD pa et Dg eS 8 Oe er ry ge hee mh eer tnd te cf oo ening + meet 4. ohn ee a ens et 


Tharsis do sow | doe ¢ Geek Gear G2 
ell bt re ¢ad Cafevesytnag if I dow't 
loot at anag ofver Guacap. Twase 
cm of he CmeRryeKy brakamag 
symm. Look esd see wien ft you cas 
fag Oeeep YOU nese 81s 6 dows 
Oat 


Ne 





Lage 52 
FAA Document 


WZ 


Be Edt View Go Commumcator Heip 


©. Back  Foxsoe Reload irene, Search szowe Prt om, oe _ 





a ep oe ce ta em — o— 


We ietiiestsoet DB inenet 2 CS Lookup 24 NewsCoot 
Community Maintenance Human 
Factors Briels. $ 


> 


Choose from the following Communities: 


sane we = . : ~ ras = =5 - sp tee men BON me et toe ee o~ : i eee ener et —— 


3 . : 2 7 ‘ 
. a+ Pip we a i ¢ 
a i 
. 


"Ef 


ENTER HERE) 


Set es Oe EN ee “ree 





Page 4 
Community Maintenance Human Factors Bnefs 


12 


R= comm brief jets - Netscape 


- . = = = _ an = 4 ‘. - =e _ at om 
Fie Edt View Go Communicator Help » . ae rT oe eet kn “TLE, 








































































coe apy = too oe as ay * Se Eo Se eR 0 a ON eae HS mich a Foe aa rae “ati presen Stee ee emma ere 
; - t a, ca . We - ” , ® « + « 1y . a ear a 0 Sa x 
= ne rs : 0 ? = “ Pad af . 4 Liye igs Fs 
= ae : ai 7, “a az : 
+4 ; aes - : ~ 4 ae a? Ce the - 4 8 t ‘ 
ee “ieee : ‘S 5 ae a4 Be’ a "a itd s . & e 
+ ‘ F .e - at weed « oe 5 T o's 

. ? ’ 5 2 - 
% Dasord H ‘ v 
of Fi . rf 
ar lation ome ‘ * ig he Be. a es | 
= ok pes Ne : os A . > ‘ wre g i 
are . 7h ~~ pee + Owe rey 
Z Related 
3 : What's = 
= in 
SN ebm ae SD ee a ee OE er rE A TT a -_ = 

a ys os ‘ nt Me ae acl oe s 










| Rinstart Message [E) intemet (Ff Lookup’ 


@ 


Monts wa es ss . 
aloe oe ven oh . 2 rte ag Files % J eee 
ry» NewsCool - Lieeta a Ce 
‘~ ” . . 
~_ ~ ow r * }. 
1 are FAL a 
hy ase Puede foes 
é . = 


~, 


bad 


ee oof. PE 
Ay 


4. 


poet ESD Pee 

| cs LU, - pM 
mt Sate ct nA 
eee 


e 


@psinsas 


ak 






ARLE aN OCse he ; Sor ane ee es aes 
ere vin? aes: ay: i : we ape’ S- 
Sheen tbe oe he’ . , Se aeneaetils 


_. [Document Done oo xs 


Sarat Center orem = 









’ Sa ewe ‘ fear ae 5 
25m Peg ~ ‘ zs < 
2S oR see eee Som. a ee e . a ee 


Page 4.1] : 
Community Human Factors Briefs for Jets 


74 





== comm brief props - Netscape 
Bie Edt View Go Communicator Help _—>- —_ : 


v aNY ” ‘ as ps * " 

~- & @ “Ais se so Sf 8B 
Pi. . 2 he - T . a 
Abe Home 


A ft Reload 









~ Use the back button to contnue your tour of this website 





_ . jDocumernt Dome 


be a Rhewe core oe Fhe ste eich we. te Neo o Fit CT INN 4 .AA AA 


Page 4.2 
Community Human Factors Bnefs for Props 


75 


= comm brief helos - Netscape 


oe ete Sis Gi twp NewsCool he ove 








= 

Community Bris for Helos will be located on this- page” ier, 
- 4 

{ 

: 

| 

: = : 

’ | 

| 

ras 2 | 

3 | | 
- 

| 

i 

; | | 

Use the weerg Paton 6 continue tour of the website’ ; | 

| 

{ 

| 








2. {| Document Done, a an Se et cae 


sm te 


ba 





Page 4.3 
Community Human Factors Bnefs for Helos 


76 


APPENDIX B. INSTRUCTION SHEET FOR SUBJECTS 


Instructions for viewing the 
Aviation Maintenance Safety Web site 


These instructions will assist in viewing the Aviation Maintenance Safety Web site. 


Computers to use: 


In the ASO Ready Room 3” Deck 


Computer names: Prowler 
Harrier 
In the Computer Lab 3 Deck 
Computer names: Skywarrior 
ex ane 
Viking 
Cobra 


Accessing the Web site: 


1. Log on the computer using your group ASO account 


2. On the desktop, double click on the icon named Aviation Maintenance Safety Web 
site 





3. When the initial page of the Web site opens, utilize the function buttons within 
Microsoft Internet Explorer (Back, Forward, Stop, etc.) to “surf” through the Web site. 


Fille? soon ia- vontess( Help) ih eats pe ean a ee de i es 


Bae Se. age Eee 


ack Forward Stop al, Home Search ae Print Font Mail . E cit 
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APPENDIX C. TASK LIST 


Aviation Maintenance Safety Web Site 
And 
Online Version of the Maintenance Climate Assessment Survey (MCAS) 


Purpose: The following tasks are designed to introduce a developmental version of the 
Aviation Maintenance Safety Web Site. This Web site is being created to assist aviation 
squadron safety personnel in assessing the climate of safety within their organization, and 
to provide useful safety reference material for squadron training. 


Your input will be used to improve this Web site and maximize it's utility to the squadron 
safety officer. 


Directions: The following list of tasks will guide your "surf" of the Aviation 
Maintenance Safety Web Site. Upon completion of the tasks, a short questionnaire will 
query your experiences with the site. There is room for comments on the survey, feel free 
to make suggestions. 


Your Assistance in making this future Web site productive is greatly appreciated! 


TASK LIST 


As you complete the following tasks, make a mental note 
of any difficulties you may experience. 
. During your tour of the site, find the online version of the Maintenance Climate 
Assessment Survey (MCAS), complete the survey and submit your results using the 
submit button at the bottom of the form 


. Find the Naval Safety Center homepage through the Links provided in the Web site 


. Search the Web site as if you were planning squadron training; find the portion of the 
Web Site that has Microsoft PowerPoint presentations to download. 





. Find the area of the Web site where mishaps by aircraft community are located. 
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APPENDIX D. EXIT SURVEY 


User’s impression of the 


Maintenance Safety Web site and 


Maintenance Climate Assessment Survey (MCAS) 


Purpose: This survey evaluates a user's overal! satisfaction of the Aviation Maintenance Safety Web site, it consists of 


three parts. 


Part I: Demographic Information. This information is used to establish the user’s computer experience and determine 


the software/hardware being used in the fleet. 


Part II: User satisfaction with the Aviation Maintenance Safety Web site. This Section deals directly with participant 


experience as they browsed the prototype Maintenance Safety Web site. 


Part III: User Satisfaction with the online version of the Maintenance Climate Assessment Survey (MCAS). This 


section allows participants to comment on the online MCAS. 


Part I. Demographic Information 


1. Iam attached to a command that performs maintenance at the: 
(Choose one from the list) 


Squadron Level 
Intermediate Level 
Depot Level 

None of the Above 


[ers] 


() CL) 


2. How long have you been using an Internet Browser? 
(Choose one from the list) 


Less than one month 

] month to less than ] year 
] year to less than 2 years 
2 years or more 


Ce 


3. What Internet Browser do you normally use? 
(Select al] that apply from both columns) 


e Netscape Navigator a 
= Microsoft Internet Explorer 

AOL’s version of Internet Explorer 
Other Browser 

None 


At Work I use: At Home I use: 


ee 


4. What categories of software have you used and are familiar with? 
(Select all that apply) 


e Word Processor (MS Word, WordPerfect, etc.) 
0 Spreadsheet (MS Excel, Lotus 123, etc.) 


81] 


Netscape Navigator 


Microsoft Internet Explorer 

AOL's version of Internet Explorer 
Other Browser 

None 


Graphic Software (Corel Draw, Adobe PhotoShop, etc.) 
Electronic Mail (AOL, Eudora, etc.) 


i 


5. What computer systems do you use? 
(Select all that apply) 


CL Windows/Windows NT 
_ Macintosh 
a UNIX 
a Other 
PART Il 
USER IMPRESSIONS 


(Fill in the choice that matches your impression) 


Strongly Agree Neutral Disagree Strongly 
Agree Disagree 


My tour of the Web site 
Was very interesting a S o os 


I found the Web site 
easy to Navigate & a = & i= 


I feel the information on 
the Web site was ina 
logical form 2 eS a = = 


The information presented 
in the Web site is relevant 
to maintenance safety ie = a 


Ee) 
[ 


My overall impression of the Web site 1s: 


During my tour of the Web site, I found: 


Strongly Agree Neutral Disagree Strongly 
Agree Disagree 


_ = an 


The use of terms consistent = Z 2 J 
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The position of screen 
instructions were consistent = L 





The results obtained from 
the Web site were predictable _ 


L} 





That adequate instructions 
were provided to navigate 
the Web site a is 





I feel that: 


The information on the 
Web site is easy to 
Understand = & 


It was easy to learn to 
use this Web site 2 a 


The organization of material 
on this Web site is clear 2 


[ ] 


I could effectively use the 
information onthis Website — ia 


When fully implemented, this 

Web site will have all the 

functions and capabilities | 

would expect it to have i ma 


To improve the usability of this Web site, ] would: 


Part Ill 


Online MCAS 


Strongly Agree 
Agree 


Finding the online MCAS 

in the Aviation Maintenance 

Safety Web site was 

straight forward = = 


The instructions for 
completing the MCAS 
are easily understood 


[ 
[ 


The task of putting answers 
in the MCAS form was 
easily completed a 2 


] would make these changes (if any) to the administration of the online MCAS 


Comments 


The most negative aspects of this Web site were: 


The most positive aspects of this Web site were: 


Your participation is appreciated 
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Neutral 


Disagree 


Strongly 
Disagree 


C } 
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